Skip to content

Integrate Wave 2 gateway features - #25

Merged
ALikesToCode merged 180 commits into
integrate/gateway-wave2from
integrate/gateway-waves-1-2
Oct 9, 2026
Merged

ALikesToCode merged 180 commits into
integrate/gateway-wave2from
integrate/gateway-waves-1-2

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

This branch combines the Wave 1 integration branch (#14) with all ten Wave 2 feature branches, so they can be tested and merged together. It is based on integrate/gateway-wave2, so this diff shows only Wave 2.

PR Feature Setting
#15 Quantization and precision preferences MODEL_PRECISION_PREFERENCE
#16 Cache policy isolation RESPONSE_CACHE_POLICY_REVISION
#17 Health probe cost forecast offline script
#18 OpenAPI contract always on, authenticated
#19 Capability probes operator script
#20 Conversion fidelity diagnostics X-MultiLLM-Conversion-Report: 1 request header
#21 Three-arm evaluation and cost replay SHADOW_EVAL_NOISE_FLOOR_ENABLED
#22 Partial usage and unknown cost always on (accounting fix)
#23 Versioned prompt templates PROMPT_TEMPLATES_ENABLED, migration 0014
#24 Configuration snapshots CONFIG_SNAPSHOTS_ENABLED, migration 0015

Integration changes

  • Kept both sides of the shared registration lists that every branch appends to:
    • the package.json test:worker list;
    • the secret firewall inventory (tests/fixtures/secret_firewall_coverage.json).
  • Added a README "Opt-in gateway features" section that links each feature's document.

Before merging to main

Two D1 migrations are included: 0014_prompt_templates.sql and 0015_config_snapshots.sql. A push to main does not apply them, and /ready reports d1_schema_missing until they exist. Apply them first:

npx wrangler d1 migrations apply multillm-intelligence --remote

Tests

  • Full Python suite: 3,181 passed. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,227 passed.
  • cf:migrations:check, test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • tests/test_frontend_build.mjs fails locally because @tailwindcss/cli is not installed. It fails the same way on main.

Provider catalogs can declare a model's precision (fp32, fp16, bf16, fp8, int8, int4) and quantization scheme. MODEL_PRECISION_PREFERENCE takes an ordered JSON list of precisions and is unset by default. While unset, catalog output and auto:intelligence candidate order are unchanged.

When set, precision is recorded only from a provider's explicit catalog fields, never inferred from a model name. It breaks ties between candidates the quality and fast profiles already rank equally. It never adds a candidate, crosses a ranking group, reorders the balanced chain or changes an explicitly requested model. The intelligence policy can override the setting with its own precision_preference list. An invalid environment value is logged once and treated as unset.
RESPONSE_CACHE_POLICY_REVISION selects the response cache identity. legacy, the default (also when unset or empty), keeps today's keys, bodies and headers. v1 adds a canonical digest of the resolved route, the key's model permissions, its secret-scan mode, the retention policy and the workflow version. A policy change therefore misses instead of replaying an answer produced under different rules.

Under v1, routes whose effective plan is not fully known (free pools, cascades, health-ordered auto routes, query parameters or provider-selection headers) bypass the cache. An answer is not stored if the policy changed while it was being generated. Unknown revisions bypass the cache, and old entries are never purged, only left unreachable.
scripts/probe_cost_forecast.py reads a JSON plan (from a file or stdin) describing the probe schedule, routes and prices. It prints upper bounds on provider calls and USD per day, per month and per run, and exits nonzero when a supplied --max-calls or --max-cost-usd limit would be exceeded.

The script never runs probes or contacts providers, and does not read the application environment. A price the plan does not state is reported as null, never as zero.
GET /openapi.json returns an OpenAPI 3.1 description of the public client API: Chat Completions, Responses, Anthropic Messages and token counting, models and image generation, including the opencode and codex-easy paths. It also covers the documented MultiLLM headers and the error and SSE envelopes. It requires a dashboard login or an API key with the models scope.

services/openapi_spec.py builds the spec without reading provider configuration, and docs/openapi.json is its committed output. A test fails when the two drift. Administrator routes are excluded.
scripts/model_capability_probe.py plans small capability probes (tools, JSON output, vision and similar) for one provider and model through the gateway. By default it only prints the plan. Sending requests needs both --execute and --allow-generation, known input and output prices, and stays within request, token, cost and time limits. The gateway key comes from an environment variable and is never printed.

Each receipt records digests of the exact redacted configuration, fixtures and policy revision it measured. A result taken under a different configuration is therefore reported as stale instead of being reused.
A client that sends X-MultiLLM-Conversion-Report: 1 gets X-MultiLLM-Conversion-Fidelity and X-MultiLLM-Conversion-Fields response headers. They summarise which request and response fields a Chat, Responses or Messages translation kept, approximated, dropped or could not represent. The summary is content-free. Without the header, payloads and responses are unchanged.

POST /v1/conversion/report returns the same report for a request without dispatching it upstream. It requires an API key, respects the key's model allowlist and limits the body to 1 MiB. The translated-protocol dispatch moves to routes/unified_bridge.py, keeping routes/unified.py under 1,000 lines. Both CORS lists allow the new request header and expose the report headers.
With SHADOW_EVAL_NOISE_FLOOR_ENABLED=true, an administrator can start a run with three_arm: true. Each comparison then replays the production model a second time, so judge noise is measured on identical answers, and a challenger's win counts only when it clears that floor under a seeded bootstrap. A claim atomically reserves all nine upstream calls (three arms, six judgments) against the daily cap; if the cap cannot cover them, nothing runs. The Worker D1 validator accepts the bounded noise_floor result only while the setting is on.

scripts/replay_routing_cost.py replays stored samples and usage offline to compare what another route order would have cost. It reports unknown prices as null and never sends a request. With the setting off, sampling, judging, daily-cap accounting, stored rows and league output are unchanged.
Provider usage is now parsed into a typed observation whose input and output counts are each optional, with a provider, estimated or unknown basis. A missing count is no longer recorded as 0, and the ledger records cost_usd and cost_basis as null when a priced component is unknown. Embeddings have a measured output of 0.

SSE usage is merged per component across events, so a stream that reports input and output counts in different events (such as Anthropic message_start and message_delta) records both. For budget limits only, a priced request with partial usage is charged conservatively, with missing components filled from the request estimate. Complete usage is accounted exactly as before, and response bodies pass through unchanged.
…o integrate/gateway-waves-1-2

# Conflicts:
#	package.json
An invalid MODEL_PRECISION_PREFERENCE is logged once and treated as unset, so the opening section no longer says it fails validation.
The Worker forwards RESPONSE_CACHE_POLICY_REVISION to the Container with its other environment settings.
The forecast doc now links the capability probe command's own execution budget instead of naming an internal work item.
PROMPT_TEMPLATES_ENABLED (default off) adds admin-published, immutable template versions and a scoped render endpoint. Rendering substitutes declared {{name}} variables once, with bounded sizes and no code, file or network access. Templates are stored in the workbench SQLite database locally or in a new D1 control-state domain (migration 0014). With the flag on and the table missing, the routes return JSON 503.
…grate/gateway-waves-1-2

# Conflicts:
#	package.json
#	tests/fixtures/secret_firewall_coverage.json
CONFIG_SNAPSHOTS_ENABLED (default off) adds admin snapshots of auto-route candidate orders with a paginated diff and a confirmed apply. Apply runs as one D1 batch that checks the revision and stored route state, writes the routes, bumps the revision and appends an audit record, so a stale review returns 409. Rollback is a new snapshot. Snapshots use migration 0015 and fail closed with JSON 503 when its tables are missing.
…grate/gateway-waves-1-2

# Conflicts:
#	package.json
Adds an Opt-in gateway features section that links the document for each Wave 2 feature.
@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 458af40d-6219-499c-928a-24646bc9e2d4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T21:08:30.872286Z 58d4f6f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
🔵 In progress
View logs
multillm-proxy 6ed5e64 Oct 09 2026, 10:41 AM

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

On a slow runner the 20 ms deadline expired before the request reached the
fetcher, so the hold was released instead of kept as unknown. The fetcher
never settles, so a 1000 ms deadline still fires after handoff.
The test checks admission uses the 1000 ms routing deadline rather than the
5000 ms policy one; a 3000 ms bound still tells them apart.
…ails

An intermittent CI failure returned 502 with no log, so the assertion now
reports the provider call count and the response body.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

feat(gateway): wire the Wave 7 enterprise features together
feat(gateway): integrate the Wave 6 gateway features
feat(gateway): integrate the Wave 5 gateway features
feat(gateway): integrate the Wave 4 gateway features
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@gitguardian

gitguardian Bot commented Oct 9, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
37552414 Triggered Generic Password c8787fc worker/alert-delivery.mjs View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@ALikesToCode
ALikesToCode merged commit f33ffa7 into integrate/gateway-wave2 Oct 9, 2026
5 of 7 checks passed
@ALikesToCode
ALikesToCode deleted the integrate/gateway-waves-1-2 branch October 26, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant